Technical Debt in Software Development : Examining Premises and Overcoming Implementation for Efficient Management

Software development is a unique field of engineering: all software constructs retain their modifiability — arguably, at least — until client release, no single project stakeholder has exhaustive knowledge about the project, and even this portion of the knowledge is generally acquired only at project completion. These characteristics imply that the field of software development is subject to design decisions that are known to be sub-optimal—either deliberately emphasizing interests of particular stakeholders or indeliberately harming the project due to lack of exhaustive knowledge. Technical debt is a concept that accounts for these decisions and their effects. The concept’s intention is to capture, track, and manage the decisions and their products: the affected software constructs. Reviewing the previous, it is vital for software development projects to acknowledge technical debt both as an enabler and as a hindrance. This thesis looks into facilitating efficient technical debt management for varying software development projects. In the thesis, examination of technical debt’s role in software development produces the premises on to which a management implementation approach is introduced. The thesis begins with a revision of motivations. Basing on prior research in the fields of technical debt management and software engineering in general, the five motivations establish the premises for technical debt in software development. These include notions of subjectivity in technical debt estimation, update frequency demands posed on technical debt information, and technical debt’s polymorphism. Three research questions are derived from the motivations. They ask for tooling support for technical debt management, capturing and modelling technical debt propagation, and characterizing software development environments and their technical debt instances. The questions imply consecutive completion as the first pursued tool would benefit from—possibly automatically assessable—propagation models, and finally the tool’s introduction to software development organizations could be assisted by tailoring it based on the software development environment and the technical debt instance characterizations. The thesis has seven included publications. In introducing them, the thesis maps their backgrounds to the motivations and their outcomes to the research questions. Amongst the outcomes are the DebtFlag tool for technical debt management, the procedures for retrospectively capturing technical debt from software repositories, a procedure for technical debt propagation model creation from these retrospectives, and a multi-national survey characterizing software development environments and their technical debt instances. The thesis concludes that the tooling support, the technical debt propagation modelling, and the software environment and technical debt instance characterization describe an implementation approach to further efficient technical debt management. Simultaneously, future work is implied as all previously described efforts need to be continued and extended. Challenges also remain in the introduced approach. An example of this is the combinatorial explosion of technology-development-context-combinations that technical debt propagation modelling needs to consider. All combinations have to be managed if exhaustive modelling is desired. There is, however, a great deal of motivation to pursue these efforts when one re-notes that technical debt is a permanent component of software development that, when correctly managed, is a development efficiency mechanism comparable to a financial loan investment.Ohjelmistokehitys on uniikki tekniikan ala: kaikki ohjelmistorakenteet säilyttävät muokattavuutensa — otaksuttavasti ainakin — asiakasjulkaisuun asti. Yhdenkään projektiosakkaan tietämys ei kata koko projektia ja merkittävä osa tästäkin tiedosta karttuu vasta projektin suorittamisen aikana. Nämä ominaisuudet antavat ymmärtää, että ohjelmistokehitysala on sellaisten suunnitelupäätösten kohde, joiden tiedetään olevan epätäydellisiä—joko tarkoituksella tiettyjen projektiosakkaiden intressejä painottavia tai tahattomasti projektia vahingoittavia puutteelliseen tietoon perustuvia. Tekninen velka on konsepti, joka huomioi nämä päätökset sekä niiden vaikutukset. Konseptin tarkoitus on havaita, seurata ja hallita näitä päätöksiä sekä tuloksena syntyviä teknisen velan vaikutuksen alla olevia ohjelmistorakenteita. Edellisen kuvauksen valossa ohjelmistokehitysprojekteille on erityisen tärkeää huomioida tekninen velka sekä mahdollistajana että hidasteena. Tämän vuoksi kyseinen väitöskirja perehtyy tehokkaan teknisen velan hallinnan fasilitointiin moninaisille ohjelmistokehitysprojekteille. Väitöskirjassa tarkastellaan teknisen velan roolia osana ohjelmistokehitystä. Tarkastelu tuottaa joukon premissejä, joihin perustuen esitellään lähestymistapa teknisen velan hallinnan toteuttamiselle. Viisi väitöskirjan alussa esitettyä motivaatiota kiinnittävät ne premissit,joille ratkaisu esitetään. Motivaatiot rakennetaan olemassa olevaan teknisen velan sekä ohjelmistotekniikan tutkimustietoon perustuen. Näihin lukeutuvat muun muassa subjektiivisuus teknisen velan estimoinnissa, teknisen velan informaatiolle nähdyt päivitystaajuusvaatimukset sekä teknisen velan polymorfismi. Havainnoista johdetaan kolme tutkimuskysymystä. Ne tavoittelevat työkalutukea teknisen velan hallinnalle, velan propagoitumisen havainnointia sekä mallinnusta kuin myös ohjelmistotuotantoympäristöjen ja niiden velka instanssien kuvaamista. Tutkimuskysymykset implikoivat peräkkäistä suoritusta: tavoiteltu työkalu hyötyy—mahdollisesti automaattisesti arvoitavista—teknisen velan propagaatiomalleista. Valmiin työkalun käyttöönottoa voidaan taas edistää jos kuvaukset kehitysympäristöistä sekä niiden velkainstansseista ovat käytettävissä työkalun räätälöintiin. Väitöskirjaaan sisältyy seitsemän julkaisua. Väitöskirja esittelee ne kiinnittämällä julkaisujen taustatyön aikaisemmin mainittuihin motivaatioihin sekä niiden tulokset edellisiin tutkimuskysymyksiin. Tuloksista huomioidaan esimerkiksi DebtFlag-työkalu teknisen velan hallintaan, retrospektiivinen prosessi teknisen velan kartoittamiselle versionhallintajärjestelmistä, prosessi teknisen velan mallien rakentamiselle näistä kartoituksista ja monikansallinen kyselytutkimus ohjelmistokehitysympäristöjen sekä näiden teknisen velan instanssien luonnehtimiseksi. Väitöskirjan yhteenvetona huomioidaan, että teknisen velan hallinnan työkalutuki, teknisen velan propagaatiomallinnus ja ohjelmistokehitysympäristöjen sekä niiden teknisen velan instanssien luonnehdinta muodostavat toteutustavan, jolla teknisen velan tehokasta hallintaa voidaan kehittää. Samalla implikoidaan jatkotoimia, sillä kaikkia edellä kuvattuja työn osia tulee jatkaa ja laajentaa. Toteutustavalle nähdään myös haasteita. Eräs näistä on kombinatorinen räjähdys teknologia- ja kehityskontekstikombinaatioille. Kaikki kombinaatiot tulee huomioida mikäli teknisen velan propagaatiomallinnuksesta halutaan kattavaa. Motivaatio väitöskirjassa esitetyn työn jatkamiselle on huomattavaa ja sitä kasvattaa entuudestaan edellä tehty huomio siitä, että tekninen velka on pysyvä komponentti ohjelmistokehityksessä, joka oikein hallittuna on kehitystehokkuutta edistävänä komponenttina verrattavissa finanssialan lainainvestointiin.Siirretty Doriast

Security in agile software development: A practitioner survey

Context: Software security engineering provides the means to define, implement and verify security in software products. Software security engineering is performed by following a software security development life cycle model or a security capability maturity model. However, agile software development methods and processes, dominant in the software industry, are viewed to be in conflict with these security practices and the security requirements. Objective: Empirically verify the use and impact of software security engineering activities in the context of agile software development, as practiced by software developer professionals. Method: A survey (N=61) was performed among software practitioners in Finland regarding their use of 40 common security engineering practices and their perceived security impact, in conjunction with the use of 16 agile software development items and activities. Results: The use of agile items and activities had a measurable effect on the selection of security engineering practices. Perceived impact of the security practices was lower than the rate of use would imply: This was taken to indicate a selection bias, caused by e.g. developers’ awareness of only certain security engineering practices, or by difficulties in applying the security engineering practices into an iterative software development workflow. Security practices deemed to have most impact were proactive and took place in the early phases of software development. Conclusion: Systematic use of agile practices conformed, and was observed to take place in conjunction with the use of security practices. Security activities were most common in the requirement and implementation phases. In general, the activities taking place early in the life cycle were also considered most impactful. A discrepancy between the level of use and the perceived security impact of many security activities was observed. This prompts research and methodological development for better integration of security engineering activities into software development processes, methods, and tools.</p

Managing Technical Debt in Software Engineering

Sustainable and scalable software systems require careful consideration over the force known as technical debt, i.e., the additional project cost connected to sub-optimal technical decisions. However, the friction that software systems can accumulate is not connected to technical decisions alone, but reflects also organizational, social, ontological and management decisions that refer to the social nature and any connected social debt of software – this nature is yet to be fully elaborated and understood. In a joint industry & academia panel, we refined our understanding of the emerging notion of social debt in pursuit of a crisper definition. We observed that social debt is not only a prime cause for technical debt but is also tightly knit to many of the dimensions that were observed so far concerning technical debt, for example software architectures and their reflection on organizations. Also we observed that social debt reflects and weighs heavily on the human process behind software engineering, since it is caused by circumstances such as cognitive distance, (lack of or too much of) communication, misaligned architectures and organizational structures.The goal for social debt in the next few years of research should be to reach a crisp definition that contains the essential traits of social debt which can be refined into practical operationalizations for use by software engineering professionals in need of knowing more about their organizational structure and the properties/cost trade-off that structure currently reflects.</p

Internal interface diversification as a security measure in sensor networks

More actuator and sensor devices are connected to the Internet of Things (IoT) every day, and the network keeps growing, while software security of the devices is often incomplete. Sensor networks and the IoT in general currently cover a large number of devices with an identical internal interface structure. By diversifying the internal interfaces, the interfaces on each node of the network are made unique, and it is possible to break the software monoculture of easily exploitable identical systems. This paper proposes internal interface diversification as a security measure for sensor networks. We conduct a study on diversifiable internal interfaces in 20 IoT operating systems. We also present two proof-of-concept implementations and perform experiments to gauge the feasibility in the IoT environment. Internal interface diversification has practical limitations, and not all IoT operating systems have that many diversifiable interfaces. However, because of low resource requirements, compatibility with other security measures and wide applicability to several interfaces, we believe internal interface diversification is a promising and effective approach for securing nodes in sensor networks.</p

Diversification and obfuscation techniques for software security: A systematic literature review

Context: Diversification and obfuscation are promising techniques for securing software and protecting computers from harmful malware. The goal of these techniques is not removing the security holes, but making it difficult for the attacker to exploit security vulnerabilities and perform successful attacks.Objective: There is an increasing body of research on the use of diversification and obfuscation techniques for improving software security; however, the overall view is scattered and the terminology is unstructured. Therefore, a coherent review gives a clear statement of state-of-the-art, normalizes the ongoing discussion and provides baselines for future research.Method: In this paper, systematic literature review is used as the method of the study to select the studies that discuss diversification/obfuscation techniques for improving software security. We present the process of data collection, analysis of data, and report the results.Results: As the result of the systematic search, we collected 357 articles relevant to the topic of our interest, published between the years 1993 and 2017. We studied the collected articles, analyzed the extracted data from them, presented classification of the data, and enlightened the research gaps.Conclusion: The two techniques have been extensively used for various security purposes and impeding various types of security attacks. There exist many different techniques to obfuscate/diversify programs, each of which targets different parts of the programs and is applied at different phases of software development life-cycle. Moreover, we pinpoint the research gaps in this field, for instance that there are still various execution environments that could benefit from these two techniques, including cloud computing, Internet of Things (IoT), and trusted computing. We also present some potential ideas on applying the techniques on the discussed environments.</p

Technical debt and agile software development practices and processes: An industry practitioner survey

Context: Contemporary software development is typically conducted in dynamic, resource-scarce environments that are prone to the accumulation of technical debt. While this general phenomenon is acknowledged, what remains unknown is how technical debt specifically manifests in and affects software processes, and how the software development techniques employed accommodate or mitigate the presence of this debt.Objectives: We sought to draw on practitioner insights and experiences in order to classify the effects of agile method use on technical debt management, given the popularity and perceived success of agile methods. We explore the breadth of practitioners’ knowledge about technical debt; how technical debt is manifested across the software process; and the perceived effects of common agile software development practices and processes on technical debt. In doing so, we address a research gap in technical debt knowledge and provide novel and actionable managerial recommendations.Method: We designed, tested and executed a multi-national survey questionnaire to address our objectives, receiving 184 responses from practitioners in Brazil, Finland, and New Zealand.Results: Our findings indicate that: 1) Practitioners are aware of technical debt, although, there was under utilization of the concept, 2) Technical debt commonly resides in legacy systems, however, concrete instances of technical debt are hard to conceptualize which makes it problematic to manage, 3) Queried agile practices and processes help to reduce technical debt; in particular, techniques that verify and maintain the structure and clarity of implemented artifacts (e.g., Coding standards and Refactoring) positively affect technical debt management.Conclusions: The fact that technical debt instances tend to have characteristics in common means that a systematic approach to its management is feasible. However, notwithstanding the positive effects of some agile practices on technical debt management, competing stakeholders’ interests remain a concern.</div

Managing Technical Debt in Software Engineering (Dagstuhl Seminar 16162)

Different methods for technical debt accumulation have been discussed, but they have mainly focused on immediate accumulation. Arguably, there is also delayed accumulation, wherein the environment around static software assets changes. Here, updates to the assets no longer deliver the environments assumptions to the assets, they become detached from the environment, and we find the assets to have accumulated debt. This debt closely reminds of software legacy, as it represents assets which can no longer be subjected under the same development actions as newly created ones. In a recent multi-national survey, it was discovered that over 75% of technical debt instances’ have perceived origins in software legacy. This communicates about the close relation between the legacy and the debt concepts. While it encourages us to further explore applying established legacy management methods for technical debt, the relation also exposes challenges. Notably, we should consider if legacy  is being  merely rebranded into the more favorable technical debt. And if this is the case, how do we ensure that all aspects of the legacy instances are identified so as to convert them into fully manageable technical debt assets. Failure to do so, will result into having technical debt assets with varying levels of accuracy, and this undoubtedly hinders technical debt management efforts overall. Nevertheless, while considering these challenges, the software legacy domain should be further researched as both a commonality of technical debt instances and as a possible interface for enhancing existing management approaches.</p

2014 IEEE International Conference on Computer and Information Technology CIT 2014

Software visualization aims to provide a more human-readable interface for the various software system aspects and characteristics. As majority of the time spent on modifying software is spent on gaining an understanding of an intangible and virtual system, the area of software visualization is widely researched as a solution to this. The paper in question presents a program visualization approach that focuses on illustrating the two software modifiability characteristics of cohesion and coupling. Unlike other approaches, which provide a visual representation for precalculated values, it uses the underlying cohesion and coupling mechanics to derive the actual layout. This allows the user to perceive the entire structure that has resulted to the cohesion and coupling values present in viewed nodes. There are three distinct steps to our approach. 1) Semantic analysis is used to record the static program structure into a directed and weighted graph. 2) The graph is then laid out using force-optimization to highlight important implementation structures. Finally, 3) sub-graph separation and further visual aids are provided to aid the user in observing cohesion and coupling for specific areas. Discussed benefits for this approach include information production efficiency, the ability to quickly analyze even large software implementations and intuitiveness of the visual delivery method.</p

2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)

Noting the overwhelming speed during software development, and particularly in environments where rapid delivery is the norm, the lack of accumulated technical debt information could result in ineffective management. We introduce technical debt propagation channels in this paper to advance software maintenance research on two accounts: (1) We describe the fundamental components for the channels, allowing identification of distinct channels, and (2) we describe a procedure to identify and abstract technical debt channels in order to produce technical debt propagation models. Our propagation models pursue automation of technical debt information maintenance with program analysis results, and translation of the maintained information between existing-and currently disconnected-technical debt management solutions. We expect the immediate technical debt information to enhance applicability and effectiveness of existing technical debt management approaches.</p